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ABSIBACT 

Some observations are presented on a social 
indicators approach to statewide educational assessment. The context 
of the observations is the Oregon Department of Education model for 
educational planning. Uuch of what is said can also be applied to 
educational planning and program evaluation at the federal level pr 
in large schocl systems. For the purposes of this discussion, a 
social indicator is defined as any statistical time series on a 
quantitative variable that is measurable and time-referenced. 
One-shot measures are excluded, as are sequences of qualitative 
descriptions. Operating from the premise that agencies may be forced 
to build educational indicators from extant data collections, som$ of 
the methodological problems. inherent in this approach are discussed. 
Some possible solutions are .proposed, both short-term and long-term, 
for measurement issues such as specifications bias, construct 
validity, and political validity. Possible areas for research and 
experimentation leading to more ideal sets of educational indicators 
are described. (BC) — 
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Ove rv I ew 

This paper contains some observations on a social Indicators approach to 
statewide assessment of education. Ihe context of the observations Is the 
Oregon Department of Education model for educational planning, A plan for a 
feasibility study on using Indicators In assessment exists In full glory 
(Jaeger, 1976) and has been submitted to the Oregon Department of Education. 
Much of what I have to. say here can be applied to educational planning and 
program evaluation at the federal level or In large school systems. The 
varlf/bles and th<?, cast of characters would change, but the basic approuch, 
and the fundamental problems, v/ould not* 

Statewide assessment ts an activity about which It might be said that, 
never before has so much energy been Invested by so many for so long, with so 
little clarity and concensus on goals and desired outcomes. If a social 
Indicators approach to statewide assessment serves no other purpose. It may 
force consideration of desired outcomes and examination of the reasonableness 
of methods. 

Although the Oregon planning model has already been reviewed In some 
detail In this symposium. It will be useful to list Its elements once again. 
The Initial step In the modal Is goal setting. Long-range, statewide goals 
for elementary and secondary education are established by the State Board 
of Education. These goals are Intended to be viewed as benchmarks against 
which the relevance of all education activities In the state can be Judged. 
Educational assessment Is the second step in the planning model. The purpose 
of assessment Is to provide data for determination of the state's educational 
status on dimensions defined by the goals established In step one. In the 
third step of the planning model, the results of steps one and two are com- 
pared. In the tradition of the discrepancy evaluation model, needs are 
defined as discrepancies between goals and current status. 
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Since It Is assumed that needs will exceed available resources, the fourth 
step of the planning model consists of ranking needs In order of priority. 
In the fifth step of the model, specific objectives are established for 
needs that have been Identified for allocation of current resources. In 
the Oregon parlance, objectives are "action Items"; they specify accomplish- 
ments by particular ejements of the educational system. In order to meet 
Identified needs. In step six of the planning model, alternative plans for 
achievement of objectives are put forth and considered. The best of the 
alternative plans that are judged to be feasible Is selected In step seven. 
Resources are allocated to the activities eclfled by the selected plan In 
step eight. In step nine, activities specified In the selected plan are 
undertaken, and a best effort Is made to Implement the plan. Step ten consists 
of a discrepancy evaluation of the plan and Its Implementation, together with 
necessary modification of objectives, the plan, or resource allocation. The 
evaluation provides some of the Information necessary to estabi Ish. goals, 
and the ten-stop model Is repeated as often as necessary. 

This Ideal Ized p lanning model Is similar In Its basic assumptions, and 
its linearity, to many others. Some of Its language may be unique, but Its 
"ready, set, go" approach to action Is familiar. Whether It will work well. 
In either a normative or absolute sense. Is a testable proposition. None- 
theless, It lends rationality to statewide assessment by providing some 
reasons for assessing educational status and progress* Before proceeding to 
examine these, I would I i ke to consi der the assumptions underlying the 
Oregon planning models and the role of assessment In planning. 

Assumptions 

Beyond the Initial goal-setting step. In which the mission of the 
educational system can be formulated In the pol Itlcal marketplace, the 



planning system assumes that decision-making Is public^ rational^ empirically- 

■ - .... .,- . 

based, and a-polItlcaL In the absence of any of these assumptions, the 

planning system could not function. For example, once goals have been set 

by the State Board of Education, they are assumed to become the stimuli for 

aM decisions and activities by educators at all subordinate levels of 

government. There can be no hidden agendas or alternative goal sets If the 

planning system Is to proceed on course. So It must be assumed that all 

educators outside the State Board of Education understand the goals, accept 

the goals, and are willing to plan and conduct their activities In ways that 

will facilitate accomplishment of the goals. The system falters If any of the 

following .assumptions are violated: 

1) Decision-makers are rational, and work only to maximize the common 
good; 

2) Decision-makers are capable of responding and willing to respond 
to externally-derived data; 

3) The educational system Is Infinitely adaptive; In the face of con- 
flicting information, current activities can be modified to meet 
current needs; 

4) Freedom of Information will be accepted by all who hold decision- 
making power; "government In the sunshine" will be practiced by the 
education bureaucracy; - 

5) Both decision-makers and the public can establish common-metric 
utilities on dimensions of current educational status, trends In 
educational status, and desired outcomes In education; 

6) All constructs of Interest and value can be measured with tolerable 
precision and validity; 

7) Causal models of the relationships between resource allocation, 
governmental Action, and educational outcomes can be established 



and val I dated, 
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The Role o.f Assessment In Planning 
If the Oregon planning system for education Is to succeed, and the 
assumptions delineated above are to be satl sf led, . assessment mus+meet two 
crucial tests. First, It must provide sufficient Information to determine 
whether, and to what degree, the State Board of Education's goals are being 
achieved. Without this Information, the third step of the planning model, 
analysis of the disparity between goals and current status, cannot be com- 
pleted. This function of assessment Is largely descriptive, possibly 
feasible, and not nearly as simple as the planning model suggests* As In most 
states, Oreon's goals for education are quite global and general • To wit: 
"In preparing for the life role of Individual, every student In the 
elementary and secondary schools shall have the opportunity to develop 
skills necessary for achieving fulfillment as a self-directed person, acquire 
the knowledge necessary for achieving and maintaining physical health, 
acquire knowledge necessary for achieving and maintaining mental heal-th, 
develop the capacity for coping v/ith change through an understanding of the 
arts, etc., etc. The list goes on at this level of generality. By my count, 
there are forty-three goals altogether. 

If, through assessment, the state Is to determine whether, for example, 
every student in the e lemente.*^ and secondary schools has the opportunity 
to "develop skills necessary for achieving f ulf I I Iment as a, self-directed 
person", common agreement rr'jst be reached on the meaning of a number of 
terms, and massive construct- val I dlty problems must be solved. What Is a. 
self-dlrectpd person? What behaviors, skills, capacities and actions would 
cause us to labe I an e lementary or secondary student- a potentla I ly "fu I f I I led 
self-directed person"? Perhaps Ted Blau'swork on life qua 1 1 ty Indicators . 
will be helpful here. But suppose that we could agree on the definition of 
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a potentially fulfilled self«-dl rected elementary school student. Our 
problems would not be over. For the goal states that ''every student In the 
elementary and secondary schools shall have the opportunity to develop the 
skills necessary...". Now we must come to some agreement on what these skills 
are^ how they can be measured^ and, of greater I mpor+ance/ 'what'^elements of 
an educational exper!e^?ce provide students with the opportunity to develop 
such skills^ Here we must move, however unwilling and unwitting. Into the 
realm of causal modeling. — 

The second task for educational assessment demands cauSal modeling 
even more forcefully than does the first. Assessment must provide a^\t)as Is 
for the setting of priorities. Including allocation of resources among 
competing prognams, projects and constituencies. To determine where scarce 
resources shall vbe allocated, decision-makers must have some basis for 
determining the results their allocation decisions are likely to realize; In 
short, they must bs able to relate decisions on educational vinputs to likely 
outcom^^s. Existing research on aptitude-treatment Interactions suggests -that 
causal models wl 1 1 be camp lex, that strong main-effects- Wl 1 1 - be few and f$ir 
between, and that low-order Interactions wl 1 1 not be highly explanatory ' 
(Cronbach and Snow, 1976). The recent resurgence of^ Interest In the Carrol I 
model of 1963, and the emergence of the^WUey-Harnlshfeger-and Harnl^^^ 
WI ley models of 1975 and 1976, suggest that we^re far from settled oh what 
Input factors affect educational outcomes. All this Is not to say 
categorically that rational , I Inear^ data-bbsed educational planning models 
won't work; just to confirm, once again, that It Isn't as'^slmple as we might 

■ Can an Indicators Approach to Assessment Help?' 
' . ' There are at least as many definitions' of the term "social Indicator" as 
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"""" " there are cx>ntrlbutors to the buTgeoril rig I I terature In the flold. The social 

Indicators movement can claim literacy thousands of papers In professional 
JoumaU^ dozens of books^ and Its own three*year-o Id Journal ^ Social Indicators \_ 
Research . For 'purposes of this discussion^ an Indicator^ whether social 
' or educational ^ will be defined as any statistical tlme^serles' on a quanti- 
tative variable that Is measurable and time-referenced*. One-shot measures 
are excluded^ as are sequences of qualitative descriptions. 

In my vlew^ the serious assessment, prob I ems I've described will not be 
readily solved through appeal to the literature on social Indicators — or 
through an "Indicators approach" to statewide assessment. I don't claim to 
have studied the entire social Indicators 1 1 terature^ but I have spent about 
a month with It. It provides no gent? ra I and unique methodology. Its principal 
contributors seem almost oblivious to problems of psychometric adequacy^ 
although Land (1975) and DeNeufville (1975) consider construct validity 
problems briefly. The principal advan-^ages of approaching educational assess- 
ment from the perspective of Indicators are these: Flrst^ attention would 
be paid to trends In educational variables^ In contrast to one-shot status 
measurement. The National Assessment philosophy advocates this rpproach^ so 
the Idea Is not new to education. But good Ideas are worthy of rel teratlofi. 
Despite current resurgence of the popularity of criterion-referenced measure- 
ment, I think that most of our value judgments are Inherently norm-referenced. 
Wilt Chamberlain Is Judged to be tall not on I y because he's somewhere around 
seven feet, but because that height Is at least three standard deviations 
above the worldwld^j mean for adult males. If I tell you that the Dow Jones > 
average for 30 Industrials closed at 1000 today, those who fo I low the stock w 
market would respond positively, not because the sea has any criterion- 
referenced meaning, or practical Interpretation, but because that Index was ^ 
at 930 just a month ago. Time series provide Inherent, temporal norms, and 
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are therefore amenable to valuing. 

An Indicators approach to statewide assessment Is also likely to bring . 
about a vastly broadened range of variables to be considered. It has the 
likelihood of fostering movement from a concentration on direct or proximal 
measurement of the supposed "effects" of educational Intervention, to the 
measurement of distal variables that are a) farther along a causal chain 
extending from Inputs and Interventions to outputs, and b) closer to ultimate 
criterion variables, Spody (1977) called the latter variables "competencies" 
and the former "capacities". In a recent article on competency-based education. 

An example of a capacity would be ability to solve problGrns.„that_requI ^ 

basic arithmetic operations; a competency Is Illustrated by ability to partic- 
ipate In a municipal election. 

Use of Indicators In statewide assessment may force educators and the 
public to define what they mean by competencies. Thus expectations placed on 
the schools by the public may become clarified. This will surely be necessary 
If Indicators of progress toward educational goals' are to be defined, presented 
to the public, and accepted as measures of educational effectiveness^ • 

As In many states, the demands on a statewide assessment. system In 
Oregon far exceeded the resources that were aval I able to that system. Mary 
Ha 1 1 cdeve loped - an I nterestlhg approach to sol vl ng that p rob I em. She suggested 
that assessment of progress toward many of Oregon\s educational goals might 
be achieved through analysis of existing data aval lable both within and out- 
side the state's Department, of Education. For example,:' the branch of state' ^ ^ 
government concerned with criminal Justice might provide time series that 
could be used to assess the effectiveness of the' state?s citizenship education 
program. The plan for exploring the feaslbl 1 1 ty of an Indicators approach . 
to statewide assessment, mentioned earlier, was based on Mary's Idea. It. 
Is an elaborate plan, since the problem Is a difficult one. And Its thirty-^ 
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four steps cannot be dealt with In a ten-minute paper that has already run 
overtimed But the^ltle Ts ^^Explorlng theTTeaslSTTlty oT using existing 
data as Indicators of progress In statewide assessment", and lt*s available 
from the Oregon Department of Education at cost of reproduction. If the 
Idea Interests you> I commend the plan to your attention. 

What I've said here may sound highly critical of Oregon's planning and 
assessment system. That view requires perspective. First, I don't have a 
better approach to offer. Second, In comparison to the activities of other 
states, the Oregon system fares well indeed. It Involves all levels of the 
education system. Including classroom teachers and the public. It defines 
the purposes-~oi education far more broadly than minimal competencies In 
basic skills. It Is adaptive, rational and open* But It will be far more 
difficult to carry out than the State Board of Education or others In the 
state government seem to realize. And an Indicators approach to assessment 



won 



't solve Its most serious problems. 
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